{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow Image Recognition  Tutorial\n",
    "\n",
    "This tutorial shows how we can use MLDB's [TensorFlow](https://www.tensorflow.org) integration to do image recognition. TensorFlow is Google's open source deep learning library. \n",
    "\n",
    "We will load the [Inception-v3 model](http://arxiv.org/abs/1512.00567) to generate descriptive labels for an image. The *Inception* model is a deep convolutional neural network and was trained on the [ImageNet](http://image-net.org) Large Visual Recognition Challenge dataset, where the task was to classify images into 1000 classes.\n",
    "\n",
    "To offer context and a basis for comparison, this notebook is inspired by [TensorFlow's Image Recognition tutorial](https://www.tensorflow.org/versions/r0.7/tutorials/image_recognition/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing pymldb and other imports\n",
    "\n",
    "The notebook cells below use `pymldb`'s `Connection` class to make [REST API](../../../../doc/#builtin/WorkingWithRest.md.html) calls. You can check out the [Using `pymldb` Tutorial](../../../../doc/nblink.html#_tutorials/Using pymldb Tutorial) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pymldb import Connection\n",
    "mldb = Connection()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading a TensorFlow graph\n",
    "\n",
    "To load a pre-trained TensorFlow graphs in MLDB, we use the [`tensorflow.graph` function type](../../../../doc/#/v1/plugins/tensorflow/doc/TensorflowGraph.md.html).\n",
    "\n",
    "Below, we start by creating two functions. First, the `fetcher` function allows us to fetch a binary blob from a remote URL. Second, the `inception` function that will be used to execute the trained network and that we parameterize in the following way:\n",
    "\n",
    "- **modelFileUrl**: Path to the Inception-v3 model file. The `archive` prefix and `#` separator allow us to load a file inside a zip archive. ([more details](../../../../doc/#builtin/Url.md.html))\n",
    "- **input**: As input to the graph, we provide the output of the `fetch` function called with the `url` parameter. When we call it later on, the image located at the specified URL will be downloaded and passed to the graph.\n",
    "- **output**: This specifies the layer from which to return the values. The `softmax` layer is the last layer in the network so we specify that one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [201]>\n",
      "<Response [201]>\n"
     ]
    }
   ],
   "source": [
    "inceptionUrl = 'http://public.mldb.ai/models/inception_dec_2015.zip'\n",
    "\n",
    "print mldb.put('/v1/functions/fetch', {\n",
    "    \"type\": 'fetcher',\n",
    "    \"params\": {}\n",
    "})\n",
    "\n",
    "print mldb.put('/v1/functions/inception', {\n",
    "    \"type\": 'tensorflow.graph',\n",
    "    \"params\": {\n",
    "        \"modelFileUrl\": 'archive+' + inceptionUrl + '#tensorflow_inception_graph.pb',\n",
    "        \"inputs\": 'fetch({url})[content] AS \"DecodeJpeg/contents\"',\n",
    "        \"outputs\": \"softmax\"\n",
    "    }\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring an image\n",
    "\n",
    "To demonstrate how to run the network on an image, we re-use the same image as in the Tensorflow tutorial, the picture of [Admiral Grace Hopper](https://en.wikipedia.org/wiki/Grace_Hopper):\n",
    "\n",
    "<img src=\"https://www.tensorflow.org/versions/r0.7/images/grace_hopper.jpg\" width=350>\n",
    "\n",
    "The following query applies the `inception` function on the URL of her picture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>softmax.0.0</th>\n",
       "      <th>softmax.0.1</th>\n",
       "      <th>softmax.0.2</th>\n",
       "      <th>softmax.0.3</th>\n",
       "      <th>softmax.0.4</th>\n",
       "      <th>softmax.0.5</th>\n",
       "      <th>softmax.0.6</th>\n",
       "      <th>softmax.0.7</th>\n",
       "      <th>softmax.0.8</th>\n",
       "      <th>softmax.0.9</th>\n",
       "      <th>...</th>\n",
       "      <th>softmax.0.998</th>\n",
       "      <th>softmax.0.999</th>\n",
       "      <th>softmax.0.1000</th>\n",
       "      <th>softmax.0.1001</th>\n",
       "      <th>softmax.0.1002</th>\n",
       "      <th>softmax.0.1003</th>\n",
       "      <th>softmax.0.1004</th>\n",
       "      <th>softmax.0.1005</th>\n",
       "      <th>softmax.0.1006</th>\n",
       "      <th>softmax.0.1007</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>_rowName</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>result</th>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000032</td>\n",
       "      <td>0.000055</td>\n",
       "      <td>0.000036</td>\n",
       "      <td>0.000047</td>\n",
       "      <td>0.000047</td>\n",
       "      <td>0.00002</td>\n",
       "      <td>0.000045</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000071</td>\n",
       "      <td>0.000082</td>\n",
       "      <td>0.00015</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000067</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 1008 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          softmax.0.0  softmax.0.1  softmax.0.2  softmax.0.3  softmax.0.4  \\\n",
       "_rowName                                                                    \n",
       "result       0.000067     0.000032     0.000055     0.000036     0.000047   \n",
       "\n",
       "          softmax.0.5  softmax.0.6  softmax.0.7  softmax.0.8  softmax.0.9  \\\n",
       "_rowName                                                                    \n",
       "result       0.000047      0.00002     0.000045      0.00006     0.000023   \n",
       "\n",
       "               ...        softmax.0.998  softmax.0.999  softmax.0.1000  \\\n",
       "_rowName       ...                                                       \n",
       "result         ...             0.000071       0.000082         0.00015   \n",
       "\n",
       "          softmax.0.1001  softmax.0.1002  softmax.0.1003  softmax.0.1004  \\\n",
       "_rowName                                                                   \n",
       "result          0.000067        0.000067        0.000067        0.000067   \n",
       "\n",
       "          softmax.0.1005  softmax.0.1006  softmax.0.1007  \n",
       "_rowName                                                  \n",
       "result          0.000067        0.000067        0.000067  \n",
       "\n",
       "[1 rows x 1008 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazingGrace = \"https://www.tensorflow.org/versions/r0.7/images/grace_hopper.jpg\"\n",
    "\n",
    "mldb.query(\"SELECT inception({url: '%s'}) as *\" % amazingGrace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is great! With only 3 REST calls we were able to run a deep neural network on an arbitrary image off the internet.\n",
    "\n",
    "\n",
    "## *Inception* as a real-time endpoint\n",
    "\n",
    "Not only is this function available in SQL queries within MLDB, but as all MLDB functions, it is also available as a REST endpoint. This means that when we created the `inception` function above, we essentially created an real-time API running the *Inception* model that any external service or device can call to get predictions back.\n",
    "\n",
    "The following REST call demonstrates how this looks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://localhost/v1/functions/inception/application?input=%7B%22url%22%3A+%22https%3A%2F%2Fwww.tensorflow.org%2Fversions%2Fr0.7%2Fimages%2Fgrace_hopper.jpg%22%7D\n",
      "\n",
      "<Response [200]>\n",
      "\n",
      "Shape:\n",
      "(1, 1008)\n"
     ]
    }
   ],
   "source": [
    "result = mldb.get('/v1/functions/inception/application', input={\"url\": amazingGrace})\n",
    "\n",
    "print result.url + '\\n\\n' + repr(result) + '\\n'\n",
    "\n",
    "import numpy as np\n",
    "print \"Shape:\"\n",
    "print np.array(result.json()[\"output\"][\"softmax\"][\"val\"]).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpreting the prediction\n",
    "\n",
    "Running the network gives us a 1008-dimensional vector. This is because the network was originally trained on the Image net categories and we created the `inception` function to return the *softmax* layer which is the output of the model.\n",
    "\n",
    "To allow us to interpret the predictions the network makes, we can import the ImageNet labels in an MLDB dataset like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [201]>\n"
     ]
    }
   ],
   "source": [
    "print mldb.put(\"/v1/procedures/imagenet_labels_importer\", {\n",
    "    \"type\": \"import.text\",\n",
    "    \"params\": {\n",
    "        \"dataFileUrl\": 'archive+' + inceptionUrl + '#imagenet_comp_graph_label_strings.txt',\n",
    "        \"outputDataset\": {\"id\": \"imagenet_labels\", \"type\": \"sparse.mutable\"},\n",
    "        \"headers\": [\"label\"],\n",
    "        \"named\": \"lineNumber() -1\",\n",
    "        \"offset\": 1,\n",
    "        \"runOnCreation\": True\n",
    "    }\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The contents of the dataset look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>_rowName</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>komondor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>racer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>524</th>\n",
       "      <td>wall clock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>278</th>\n",
       "      <td>snowplow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>211</th>\n",
       "      <td>German shepherd</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    label\n",
       "_rowName                 \n",
       "97               komondor\n",
       "273                 racer\n",
       "524            wall clock\n",
       "278              snowplow\n",
       "211       German shepherd"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mldb.query(\"SELECT * FROM imagenet_labels LIMIT 5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The labels line up with the *softmax* layer that we extract from the network. By joining the output of the network with the `imagenet_labels` dataset, we can essentially label the output of the network.\n",
    "\n",
    "The following query scores the image just as before, but then transposes the output and then joins the result to the labels dataset. We then sort on the score to keep only the 10 highest values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>_rowName</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>military uniform</th>\n",
       "      <td>0.808008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>suit</th>\n",
       "      <td>0.022845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>academic gown</th>\n",
       "      <td>0.009540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bearskin</th>\n",
       "      <td>0.009413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pickelhaube</th>\n",
       "      <td>0.007743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bolo tie</th>\n",
       "      <td>0.006794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bulletproof vest</th>\n",
       "      <td>0.005836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bow tie</th>\n",
       "      <td>0.004035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cornet</th>\n",
       "      <td>0.003984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Windsor tie</th>\n",
       "      <td>0.003208</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     score\n",
       "_rowName                  \n",
       "military uniform  0.808008\n",
       "suit              0.022845\n",
       "academic gown     0.009540\n",
       "bearskin          0.009413\n",
       "pickelhaube       0.007743\n",
       "bolo tie          0.006794\n",
       "bulletproof vest  0.005836\n",
       "bow tie           0.004035\n",
       "cornet            0.003984\n",
       "Windsor tie       0.003208"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mldb.query(\"\"\"\n",
    "    SELECT scores.pred as score\n",
    "    NAMED imagenet_labels.label\n",
    "    FROM transpose(\n",
    "        (\n",
    "            SELECT flatten(inception({url: '%s'})[softmax]) as *\n",
    "            NAMED 'pred'\n",
    "        )\n",
    "    ) AS scores\n",
    "\n",
    "    LEFT JOIN imagenet_labels ON \n",
    "        imagenet_labels.rowName() = scores.rowName()\n",
    "\n",
    "    ORDER BY score DESC\n",
    "    LIMIT 10\n",
    "\"\"\" % amazingGrace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Where to next?\n",
    "\n",
    "You can now look at the [Transfer Learning with Tensorflow](../../../../doc/nblink.html#_demos/Transfer Learning with Tensorflow) demo.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
